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ABSTRACT 

This paper reviews what has been learned about 
evaluation ultilization during the past 20 years. Evaluation 
utilization is discussed in terms of what is used, who uses 
Evaluation, when evaluation is used, how evaluation is used, vjhere , 
evaluation is used, and why evaluation is used. It is suggested that 
the personal factor - the interests and commitments of key people 
involved in the evaluation - is the most important explanatory 
variable in evaluation utilization. Teh Basic utilizat ion-f ociised 
evaluation premises are presented, and it is sjiggeste'd that the 
primary direction * for further utilization 'research is to test these 
premises in an integrated, holistic way in real^ evaluations. The 
recent external fcvalu&tion of the Caribbean, Agricultural Extention 
Project is described to illustrate a comprehensive test of 
utilization fQcused evaluation in a real world project. Five. Lssues 
that continue to create confusion in the field and that'emerge from 
discussions of the stakeholder assumption and research on the 
personal factor are discussed. It is concluded that evaluators should 
act pn what is already known about utilization and evaluate those 
actions. ralher that do farther isolated research on utilization. 
Preliminary observations on the problems of misutilization are also 
presented. (BS) 
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. The practice of evaluation involves the systematic collection of 
^ information about the activities, characteristics, and outcomes of 

programs, ^personnel , and products for use by specific people to reduce 
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vO' uncertainties, improve effectiveness, and make decisions with regard to 
C\J 

q what those programs, personnel, or products are doing and af feet ing.y "This 
LU 

def ini tion of evaluation emphasizes (1) a systematic collection of 
information about (2) a broad range of topics (3). for use by specific 
people (4) for a variety of purposes. 

This definition of evaluation is purposefully broad and includes, 
quite explicitly, the notion that evaluation can te used .^in a variety of 
ways. Definitions of evaluation vary, with some being quite narrow. In 
reviewing research on the utilization of evaluation, it is helpful to begin 

with-how evaluation is defined because variations in definitions of 

i 

evaluation lead to variations in definitions of utilization, and make the 
task of conducting reseaVch on utilization quite complex and varied. 
Variations in Evaluation Definitions 

William J. Gephart's comprehensive effort at defining evaluation 
illustrates both the problem and one kind of solution. He begins with the 

assumption that no short, succinct definition will suffice. 

* /■? 

^* Single-sentence definitions usually contain a hbst of terms that need 

\0 * further definition to clarify the original definition. He proceeds to 

^ define evaluation in six different ways, different^in that each represents 

^ a distinct approach to, the definitional tarsk- (1) His classifica^ory 

|^ definition describes evaluation as 3 "prpbVem-sol ving strategy" employed 
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for establishing the relative or absolute worth of various choices. (2) 

Hfs comparative definition 1 i kervs~ eval uati on to research, development, 

management, and other problem-solving strategies, pointing out similarities 

and differences with each. (3) His operational definition tells how an 

evaluation is C9nducted, from identification of the impending decision 

through data^col 1 ection and analysis to information use. (4) His 

compenential definition explains theft evaluations include a problem, a 

situation involving choices, data on th*e worth of options, a context, a set 

of values, a time frame, and so on. (5) His ostensive definition gives 

examples of evaluations (e.g., deciding which dishwasher to buy). (6) His 

i ' 

synonym definition includes such words as judgment and appraisal. He 
concludes that these six definitions, "taketi together," form his concept of 
evaluation. He also notes that one of the difficulties encountered in 
sharing definitions >is that, while there are at least six different ways of 
approaching the definitional task, "most of us fall into the habit of using 
only one of tyem" (Gephart, 1981 :250-25b) . 

Gephart's effort shows that there are various ways of approaching the 
definitional task. Further complicating the problem is the fact that 
within any one or more approaches, the content of the definition can vary. 
A review of a few of the variations in the content of definitions of 
evaluation reveals important differences in what various evaluators 
emphasize in their work. * 

(1) The classic approach of Ralph Tyler (1949) was to emphasize goals 

k 

and objectives, so for him (and for the thousands of educators and 
researchers schooled in his approach), 'evaluation is the process of 
determining the extent to which the goals and objectives of a program are 
being attained. 
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(2) Many social scientists emphasize scientific rigor in their 
evaluation models, and that emphasis is reflected in their definition of 
the field* For these social scientists, evaluation involves primarily the 
application of rigorous social sci ence, methods to the study of programs 
(e.g., Bernstein and Freeman, 1975; Rossi, Freemar^, and Wright, 1979). 
These evaj uators. emphasize the importance of experimental designs and 
quantitative measures. 

(3) Another common emphasis in evaluation definitions is on the 
comparative nature of the process: Evaluation is the process of comparing 
the relative costs and benefits of two or more programs. The principles 
and definitions that undergird evaluation models emphasizing the 
comparative nature of the process have emerged in part as a- reaction to the 
narrowness 'of evaluation when defined as measuring relative attainment of a 
single program's goals (see Alkin and El lett r J984) . 

(4) Still another emphasis comes from evaluators who highlight the 

s 

valuation part of evaluation. From this perspective^ evaluation., is t^he 
process of judging a programs value. This final judgment, this ultimate 
determination of relative merit or worth, is the sine qua non of evaluation 
(see Worthen 4 and Sanders, 1973: 22-26, 120-122; Guba and Lincoln, 1981: 
35-36). 

(5) Some evaluation practitioners focus on the generation of data for 
decision making'and problem solving. This perspective goes beyond making 
judgments or assigning relative values. The etaphasis is on choices, 
decisions, and problem resolution. It is quite possible to decid^ that one 
thing is better than another (e.g., prograip versus program Y) without? 
taking any concrete decision with regard* to. pfograiji X Qr .program Y. When 

4 1 * 

evaluation is defined as a problem-sol vlAg process- (Geghart, 1981) or as a 

\ 
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process that provides information for decision making (Thompson, 197b), 
some action process that goes beyond valuation is given primary emphasis in 
the definition. < 

(6) Finally, for the purposes of this discussion, there are those 
definitions tha^ emphasize providing information/to specific people. The 



broad definition I use most often takes this;appfoach. 



The practice of evaluation involves the systematic colVectron 
of information about the activities, Characteristics, and' j 
outcomes of programs, personnel* and products for use by 
speci fic ^people to reduce uncertainties, improve effectiveness, 
and make decisions with regard to- what those programs, personnel 
or products are doing and affecting. This de'finiticn of evaluation 
emphasizes (1) the systematic collection of information about 
(2) a broad range of topics (3) for use by specific people (4) for 
a variety of pu^pp^es. 

This definition is the basis for a "usTr^focused" approach tg evaluation 

(Patton, 1981: 83-89), wh.ich places emphasis on the information needs and 

Interests of specific people, such needs including, but not limited to, 

irvformati ort relevant to making decisions, judgments, comparisons, or goal 

attainment assessments.,, 

Now then, we have six different types of evaluation definitions 

{classif icatory , comparative, operational, componential , ostensive, and 



synonym) and six different emphases in various definitions. (goals, methods, 
comparisons, value, decisions, and information users). Nor do these cover 
all the possibilities. For example, in the study of how evaluations are 
yjsed that formed the basis for Utilization-Focused Evaluation (Patton, 
1978), I began with a collection of 170 "evaluations" on file in the Office 

of Health Evaluation. Fewer than half of those-170 federal health studies 

* * * 

could^ be considered "evaluations" using any of the definitions just 

> 

reviewed. This was because a large number of those studies were 
nonempirical think pieces (i.e., they, intl uded no systematic data • 
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collection or analysis) or they focused on general social indicators 
without reference to any specific program. Still, they were filed 
(defined!) as evaluations. 

Let me now make several observations based on the preceding 
discussion. First, no single-sentence definition will suffice to fully 
capture the practice of evaluation. Second, different definitions serve 
different purposes, one especially important function being to serve as a 
foundation far a particular model of or perspective on evaluation. Third, 
there are fundamental disagreements within the fi^ld about the essence and 
boundaries of evaluation. Fourth, people who propound a particular 
.definition often have some ego investment in their special perspective, 
whether because they developed it, were trained according to it, or are 
part of a group in which that definition is esteemed-^ any critique of a 
definition, in s.uch cases, can be taken as 3 personal attack, a good many 
people finding it difficult to separate criticism of, their ideas from 
criticism of them personally. Fift^i, people on the outside \ooking in (and 
many within* the field) are often confused and, uncertain about just what 
^valuation is./ Sixth, there is no reason to expect an early end to either 
the disagreements or the confusion. As Samuel Butler explained the problem 
in "Higgledy-piggledy," 

) 

Definitions are a kind of scratching and generally leave a 
sore place more spre than it was before. ' 



UTILIZATION 



Since there is no universally excepted definition of evaluation, there 
can be no universally accepted definition of utilization . Any given 
definition of utilization will necessarily be dependent on and is derived 
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from a prior definition of evaluation, whether that definition is implicit 
or explicit.. "As Eleanor Chelimsky has written: ; "The concept of 
useful-ness. ..depends upon the perspective and values of the observer. This 
means that one person 1 s .useful ness may be another person 1 s waste" 
(1983* 155 ) % . 

It is helpful to keep these definitional variations in mind as we 

review what we have learned about utilization during the last twenty years 

— and I believe we have learned a great deal about the'Niti 1 ization of 

evaluation.. Discussions about and research on utilization have contributed 

to the emergence of "utility" as one of the four central themes in the 

standards for evaluation developed by the JoinV Committee on Standards for 

Education Evaluation (1981). At -the professional meetings of the* 

Evaluation Research Society and t}ie Evaluation Network concerns about 

increasing the utilization of evaluation hav6 been pervasive. We know, 

therefore, a great deal more than we knew a few years ago. I want to first 

review some of what we know and "then turn to the future and identify areas 

about whjch we-need to know more. To organize this discussion about what 

we know I shall use the six honest serving men of Kipling: 

\ keep six honest serving men. 
They/taught me all I knew: 
Their names are What and Why and When 
And How and Where and Who. 

What Is Use 

Evaluations can have conceptual or action impacts. Conceptual impacts 
are those which affect thinking about a program. Such uses may lead to 
conceptualizing implementation or outcomes in new ways, understanding 
dynamics of the program more thoroughly, or shifts in program priorities. 
Action Impaots are those which lead to"* observable changes in the actual , 4 

operations of a program. These are most notable and dramatic when they 
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involve changes in levels or types o^^fnrratn^/or changes in program 
deli-very. Evaluations can also affect decisions. Evaluation may lead to' a 
decision to continue or stop a program, or to do any of the large variety 
of things over which decisionmakers have control. A decision to do 
absolutely nothing new or different can b£ a major evaluation impact but 

will not lead to any observable action or change as a result of the 

C 

evaluation. 

v The research on utilization is typically biased towards action 
impacts. The early litanies about the lack of utilization of evaluation 
were based on narrow definitions^^ utilization limited to immediate 
action. It is clear, though, from discussions with people who actually use 
evaluations (Patton, 1978) that reinforcing or challenging ways of thir.king 

\ 

are important impacts for decisionmakers attemncing to reduce their 
uncertainties about grograms. 

The rel at ive" importance, then, of eval uat ion jjfti 1 i zation can only be 
judged perceptually by the value attached to utilization by those who use 
the evaluation. There can be no absolute standard which values action over 
thinking, changes in a program pver keeping things the same, or decisions 
to do something over decisions to wait. There simply can be no hierarchy 
of impacts because the hierarchy is necessarily situational and depends on 
the values and the ,needs of the people for whom the evaluation is 
conducted. 
What Is Used 

, — * 

/ 

Early research on utilization focused on the outcomes of evaluation, 

-i.e., the data, the recommendations, and the evaluation report. When the 

J 

question was^asked, "Was the evaluation useful?" the implicit assumption 
was that one was talking about the findings and recommendations of the 



eval uation. 

As our ynderstdrtdiny of the utilization process has ^ric-t^ased, 
however, we have come to understand that evaluation processes can have 
significant impact quite apart from the outcomes of the evaluation. 
Indeed, evaluation processes can be used even if there are no outcomes of 
an evaluation, for example, if data collection falls apart and no report is 
ever written. Evaluation processes can be useful in helping program staff 
clarify what they, are doing, establish priorities, focus resources and 
activities on sp_ecififc outcomes, and identify areas of weakness even before 
data are collected. Evaluation processes are useful because they stimulate 
staff to think rigorously about their program in ways which might not 
happen without t{ie fbrced stimulus of coming to grips with the demands of , 
the evaluation. 

In addition to the use of evaluation findings and evaluation 
processes, there is the use of the evaluator. Quite apart from 
facilitating evaluation processes and producing outcomes, the evaluator can 
be useful in program development and decision-making as a professional 
consultant who is sensitive to and insightful about the program. This 

utilization of the evaluator can be viewed quite apart from and go well 

X. 

beyond the more narrow confines of data collection and goals clarification. 

» * 

It is not unusual for evaluators to be asked for their perceptions, their 

4 

impressions, their managerial assistance, and, their general advice above 
and beyond the narrow focus of dat? collection. SQme evaluators are 
uncomfortable with this larger roie and refuse to take it on, but many 
evaluators consciously or unknowingly become management consultants and 
important advisors to programs. 



Wh o Uses Evaluat ion , N 

There are multiple and varied interests around any evaluation. 

Teachers, admi nistjT&tors , parents , v publ ic officials, funders, students, and* 

community leaders all have an interest in educational evaluations, but the 

degree and nature of -their interests will Vary^ We have learned that these 

different constituencies use evaluations in different way?. Program staff 

are most likely to benefit from utilization of th£ evaluation process. ^ 

Funders and the community people are most likely to use published data and 

written 'findings. Administrators kre most likely to use the eyaluatcr as a 

consultant. The kind (^/impact also varies. An evaluation is likely to be 

used to affect the thinking and conceptual i zatian of people more distant 

from the day-to-day operations of the' program, tt> affect actions taken by 

those actually involved in the day-to-day delivery of the program, and to 

affect the decisions tcfken by those with overall responsibility for the 

program, i.e., funders and administrators. 

We have also learned that the extent to which these various % 

constituencies are well served will vary* from evaluation to evaluation. .No ^ 

single evaluation is likely to be able to serve all constituencies equally 

well. Either implicitly or explicitly the evaluation design includes bias 

toward the information and process needs of some constituencies more than 

others. We have also learned that we should include ajnong the list of 

possible constituencies who benefit from the evaluation the evaluator 

t * 
himself or herself. An evaluation may serve the needs and interests of the 

evaluator more than those of any other constituency. Whether such use is 

acceptable, justifiable or important depends on where one is sitting and 

what values are brought to bear on the question of utilization. Nor am I 

talking here simply about lining the pockets of evaluators or providing 



academios with publishable papers." I recently talked wjth a state 
administrator of welfare programs who 'had created an internal evaluation 
unit at the state level. During the first year of operations,^ the 
evaluation unit had conducted several evaluations all of which the state 
administrator judged to be useful "because tftey helped the evaluators learn 
how to conduct evaluations at the state level*" None of these evaluations 
had yielded particularly useful information for the state administrator, 
nor were the subjects of the evaluation important for the state. The 
purpose of the evaluations conducted during the first year were entirely 
aimed at making^the unit operational so that ;jt could be useful in 
generating important data for decision-making in the future. This is a 
case of utilization of evaluation processes for the benefit of evaluators, 
at least in the short run. 
When Is Evaluation Used 

The qualifying phrase at the end of tji^ last sentence points out the 
problem of determining a time horizon for the utilization of evaluations. 
The early literature on- uti 1 ization of evaluations focused on i tinned i ate 
action' impacts. Subsequent research found 'that evaluation utilization was 
more likely to be incremental than immediate. This means that, in many 
cases, evaluation processes make a difference time and that evaluation 
outcomes (findings) are discussed and used over a, period of time. This 
.incremental nature St % eval uation utilization flows in part from the 
incremental nature of most decision making. There are not a great many 
clear, specific and immediate decisions taken in public organizations. 1 
Rather, decision making tends to be a process of moving in a particular 
direction that* is not always explicit and does not always come from 
decisive moments of action. There remains, I believe, a bias in the 
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research literature on utilization in the direction ot preferin^ ipwumm: 
concrete and short-term impacts to more diffuse and longer term unpads. 
This is partly i ri response to the measurement problem, i.e., that more 
immediate impacts are edsier*to yet at and are more visable. However, 
incremental impacts over a lonjyer period of time may be more important lf < 
many cases. 

How Is E valuation Used2 

There are many dimensions one might consider here. I want*to tucus ;^ 
two, more by way of example than because they are definitive, although they 
have been particularly important in the evaluation utilization literate. 
Evaluation utilisation can be planned or unplanned*, and can be formal or 
informal. Planned utilization occurs when the intended use of the 
evaluation is identified at the beginning and then subsequent utilization 
follows and is judged by planned or intended use. Unplanned utilization 
occurs when, in the typical case, the evaluation is designed y/ithout 
particular attention to questions of utilization and questions of jse are 
left until the data are collected and analyzed. Eleanor Chelimsky argues 
that the most important kind of accountability in evaluation is utilization 
that comes from "designed tracking and rol low-up of a predetermined use to 
predetermined user" (1983:160). Chelimsky calls this a "closed-looped 
feedback process" where "the policy makeoTwants information, asks for it, 
and is interested in and informed by the response" (1983: lbUh f rom this 
perspective, the most important question in researching the utilization of 
evaluation is whether the evaluation had its intended use. This solves the 
problem of defining utilization, addresses the question of who the 
evaluation is for, and builds in a time frame since the predetermined use 
would necessarily have a time frame. 
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The problem here is the same problem that emerges in evaluation itself 
When attention is directed only to the stated and explicit goals of the 
project.* The debate that led to Scriven's proposal for goal free 
evaluation included concerns about attention to unanticipated consequences, 
side effects, and unstated goals as important outcomes of programs. 
Attention only to explicitly stated goals would miss these other impacts. 
Likewise, in looking at utilization, limiting attention to the explicitly 
stated expectations for utilization will miss longer term, unintended, and 
unplanned uses, any of which may be quite important. 

Another aspect of how evaluations are used that has become important 
is whether uses formal or informal. The early research on utilization 
foe ised on formal uses, that is public, observable, and explicit uses of 
published findings. We have since learned that informal uses are often 
more important,. This is the transfer of findings by word of mouth, in 
unplanned discussion groups, and in one-to-one interactions between the 
evaluatx)r and program staff, administrators, and/or funders. Such informal 
interactions often go well beyond official evaluation findings, and it is 
in the informal process of utilization that the evaluator himself or 
herself is likely to be used as much as or more than efther the formal^ 
evaluation process or findings. 
Where Is Evaluation Used? 

The problem of where evaluation is used has emeryed most directly in 

efforts at satisfying the different needs of evaluation users at the local, 

state, and national levels in education. Framed in this way the question 

of where is closely related to the question of who. But the question of 

where the eva-rUatpon is used is a larger dimension in that evaluation 
/ 

designs and potential uses at the national level are quite different from 

\ 

/ - 12 13 



those aimed at local utilization. In a perfect world, the kind typically 

demanded by political rhetoric, a single evaluation would be useful at all 

levels from the local school up through the national government. In 

reality, the information needs of these different units are dramatically 

different* Indeed, the most common problem I encounter in evaluation 

consulting is dealing with the conflicting information needs of people at 

different levels of government. '<The state system imposes data collection 

requirements on local units that they preceive to be usejess while data 

collected entirely by local initiative seldom meets the needs of either 

state or federal governments. Local units tend to prefer highly 

♦ 

idiosyncratic and situational ly specific data. Larger units ten£ to prefer 
standardized data which makes aggregation and comparisons easier. All of 
the dimensions of utilization vary according to where the evaluation is 
used. fOne of the greatest challenges for evaluations that are part of 
management information systems is responding to utilization needs at these 
di f ferent 1 evel s. 

Why Is Evaluation Used? ' 

The "why" of evaluation use has focused most often ofTthe distinction 
between formative and summative evaluations. Indeed, the classic' 
formati ve-summati ve distinction was intended to define different kinds of 
evaluation use, i.e., evaluations aimed at program development and 
improvement, versus evaluations aimed at major go/no-go decisions and/or 
major funding decisions. In practice, however, the "why" question is 
considerably more complex than this. The reasons evaluations are used, or 
not used, run the gamut of human motivations and schemes. There are highly 
political reasons why evaluations are used or" not used. There are 
personality dimensions to this problem. There are personal value reasons, 
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and matters of personal integrity and motivation. There are reasons having 

to do with human factors, context factors, and characteristics of the 

evaluation. Indeed, the question of why evaluations are used leads 

directly to the research literature which reports on the factors which 

affect evaluation use, which explain utilization, and which describe 

varying conditions under which utilization takes different forms. Indeed, 

most of the res earch on utilization has focused on identifying the factors 

that contribute to use rather than on variations in utilization itself , 

this later point having been the focus of my discussion thus far. 

James'Burry (1984) has done a thorough review of the evaluation 

utilization literature aimed at a synthesis of factors which appear to have 

a bearing on the degree to which evaluation information may be used. He 

organizes the various factors in three major catagories: human factors, 

context factors, and evaluation factors. 

Humar\ factors reflect evaluator and user characteristics 
with a strong influence on use. Included here are such 
factors as people's attitudes toward and interest in the 
program and its evaluation, their backgrounds and organizational 
positions, and' their professional experience levels. 

Context factors cqnsist of the requirements and fiscal restraints 
facing the. evaluation, and relationships between the program being 
evaluated and other segments of its broader organization and the 
surrounding community. 

Evaluation factors refer to the actual conddct of the evaluation, 
the ftrocedures used in the conduct of the evaluation, and the 
quality of the information it'provides*. (Burry, 1984). 

Since I presume that the participants in this meeting will have access 

to the Burry review, I shall not discuss it in greater detail at this time. 

That review, in conjunction with and as a part of the framework developed 

by Alkin jit al^ (1979), presents a comprehensive look at the factors 

affecting evaluation. (In accepting the invitation to do this paper I had 

orginally expected to do what I subsequently found James Burry had done. 
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His work is so well done I saw no reason to replicate his efforts.) 

The primary weakness of the synthesis and of the framework developed 
by Alkin et a]_ (1979) is that the factors are undifferentiated in terms of 
importance. The synthesis represents a checklist of factors which can 
influence evaluation, and the literature which is synthesized suggests the 
conditions under which certain factors will emerge as important, but no 
overall hierarchy is suggested by the synthesis, i.e., a hierarchy which 
places more^importance on certain factors as necessary and/or sufficient 

conditions for evaluation utilization. In the next section I want to take 

i 

on this problem of differentiating the relative importance of various 
factors which explain utilization. 

THE PERSONAL FACTOR 

I 

I want to suggest thajt the personal factor is the most important 
explanatory variable in evaluation utilization. I make this assertion 
quite deliberately in oYdjr to be provocative. The personal factor emerged 
as the most important variable in the initial research that led to f 
Utilization-Focused Evaluation (Patton, 1978). A great dbal^ of subsequent 
research has validated the importance of this factor, and I know of no 
research which would indicate that it is not the key variable in 
util ization. 

The personal factor has to do with the interests and commitments of 
the ksy people involved in the evaluation. Where the key people are 
interested in^ committed to, and involved in the evaluation for the purpose 
of making sure that it is useful, then the evaluation is likely to" be used. 
Where those interests^ commitments and involvement are not present, 
evaluatipn is considerably less likely to be used. 

" .-15-16 



The personal factor is general in conceptualization. It includes 
several, but not all, of the factors listed in the Burry synthesis as 
"human factors." However, I prefer the term "personal factor" to "human 
factors" because the phrase, "the personal factor," is meant to explicitly 
communicate that the personal characteristics of individual people is what 
majces the difference. This is in strong contrast to structural, 
organizational, and methodological explanations. 

Identification of the personal factor as the key explanatory variable 
also provides a bridge from description to prescription. Given that the 
standards for educational evaluations (1981) have included a clear mandate 
I f or evaluator accountability wnich holds forth the ideal that evaluations 
I first and foremost should be useful, it seems to me appropriate to use our 
knowledge of factors affecting the utilization of evaluation to make 
prescriptive statements that will guide evaluators in their efforts, Such 
prescription goes beyond saying that one ought to take a certain list of, 
factors into account when designing the evaluation." A- prescri pti va 
statement that is research-based wduld tell evaluators how to take those 
factors into account. The personal factor provides such a prescription by 
paying one takes those factors into account in terms of the values, 
interests, and understandings of the people who are to use the evaluation 
results. * 

For example, I began this paper by reviewing the problem of different 
definitions of evaluation. A neutral checklist approach to evaluation 
practice ^/ould advise the evaluator to understand that tl/ere ire different 
definitions of evaluation and to be sure to define what jund of evaluation, 
is being undertaken in a particular effort. However, a ihtil i zation-focused 
.approach built on the importance of the personal factor provides more 
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direct advice about how to proceed. A utilization-focused approach begins 
by finding out the perceptions and definitions of the people with whom one 
is working. Before the evaluator unilaterally defines evaluation,, the 
eval uator should work to discover the perceptions,, confusions, 
expectations, and beliefs about evaluation of those people who will be the 

primary users of fhe evaluation. It is ahen possible to build on that 

i 

knowledge to develop Shared understandings about evaluation options and 
potential processes. . It is often appropriate simpdy to, ask the people 
with whom one is working to associate freely in a stream-of-consci ousness 
fashion with the word earticipants to define evaluation. The question, 
"Who can give me a definition of evaluation?" clearly implies a single 
right answer, and the toary participant will suspect that tfie evaluation 
trainer or facilitator will eventually pronounce the correct definition, 
but only after making several participants look stupid. Definitions are 
thus perceived as academic playthings to be used in a game at which the 
researcher is sure to win, so why participate? I'm not looking for skill 
dt constructing or repeating definitions. I'm looking for perceptions and 
synonyms that will provide clues to tacit definitions held by people in the 
\ situation in which I'm looking. With these perceptions made explicit it is 
then possible to consider other alternative^ and end by defining evaluation 
in a way that is relevant to the people who are going to use the 
eval uati on. 

In bridging the gap between description and prescription, the gap 
between a list of possible factors and the more direct identification of 
how those factors come into play, it is possible to use our knowledge of 
evaluation utilization to state a set of premises or prescriptions about 
how to increase utilization. Those premises constitute what I have called 
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"util i zat4on-focuied evaluation. " I am presenti ng, them here as a way-of 
testing the extent to which we agree that' there is a research base for 
these premises. 

i 

UTILIZATION FOCUSED EVALUATION PREMISES 

1. Basic Premises of Util i zation-Focused* Evaluation 

I 

The first premise is that concern for utilization should be the 
driving force in an evaluation. At; every point where a decision about the 
evaluation is being made, whether the decision corfcerns the focus of study, 
design, methods, measurement, analysis, or reporting, the evaluator asks: 
"How would that affect the utilization of this evaluation?" - 

The second premise is that concern for-utilfzation is<6n-going and 
continuous from the very- beginning of the evaluation. Utilization isn't 
something one becomes interested in at the end of an e\Acffuation. By the 
end of the evaluation, the potential for utilization has been largely 
determined* From the moment decisionmakers and evaluates begin 
conceptualizing the evaluation, decisions are being made which will affect 
utilization in major ways.* ,■ 

Jhevthird premise is that evaluations should be user-oriented. 'This 
means that the evaluation is aimed, at the interests and i nfohnati on needs 
of specific, identifiable people, not vague, passive audiences. Therefore, 
the first step in util ization-focused^valtiation >s identification or 

specific decision makers and iViformation users. The 
^valuator must determine who the potential users are, and aim the 
evaluation at thoseUsers. , r ^ A 

A fourth premise is that, once identified, these' interested decision 
makers aQd information users should be personally and actively involved in 
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making decisions about the evaluation. Working actively with people who 

have a stake in the outcomes of an evaluation (the "stakeholders") is aimed 

at increasing the potential for utilization by building a genuine 
* <* 

commitment to and understanding of the evaluation over the course of the 
evaluation process. Such an approach recognizes the importance of the 

* 

"personal factor" (Patton, 1978) in evaluation utilization. People who are 
personal ly interested and involved in an evaluation are more likely to use 
evaluation findings. The best way to 6e sure that an evaluation is 
targeted at the personal concerns of stakeholders is to involve them 
actively at every stage of the evaluation.' 

A fifth premise is that there are multiple and varied interests around' 
any evaluation. Teachers, administrators, parents, public pfficials, 
students, and community leaders all have an interest in evaluation, but the 

degree and nature of their interests will vary. The process of identifying 

< 

and organizing stakeholders to participate in an evaluation process should 

be done in a way that is sensitive to and respectful of these varied and 

multiple interests. 'At the same time, it must be recognized that resource, 

time, and staff limitations will make it impbssib\e for any single 

I 

evaluation to answer all possible questions, or to give full attention to 
all possible issues. Identified decision makers and information users, 
representing various constituencies, should come together at the beginning 
of the evaluation to decide which issues and questions will be given 
priority in the evaluation in order to maximize the utility, of the 
evaluation. The process of focusing the content of the evaluation should 
not be done by evaluators acting alone, or in isolation from users. 

A sixth premise is that careful selection of stakeholders for active 
participation ip'the evaluation process will permit high quality 

r 
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participation , anil high quality participation is the goal, not high 
quantity participation. The quantity of group interaction time is often 
inversely relate(jJ:o the quality of the process. Thus, evaluators 
conducting utilization-focused evaluations must be skilled group 
facilitators and have a large repertoire of techniques ava^/lable for 
working actively with stakeholders in the evaluation (Patton, 1981). High 
quality involvement of stakeholders will result in higher quality 

evaluations. Many evaluators assume that methodological rigor will 

/ 

inevitably be sacrificed if non-scientists collaborate in making methods 
decisions. This need not be the case. Decision makers want data that are 
useful and accurate (Weiss and Bucuvalas, 1980). Skilled evaluators can 
help non-scientists understand methodological issues so that they can judge 
for themselves the trade-offs involved in choosing among the strengths and* 
weaknesses of design options and methods alternatives. Such involvement in 
collaborative deliberations on methodological issues, can significantly 
increase stakeholders' understanding of the evaluation, while giving 
evaluators a better understanding of stakeholder priorities and situational 
constraints on the feasibility of alterna\ive approaches. These^ shared 
decisions can thus enhance both utilization potential and methodological 
rigor. 

A seventh premise is that evaluators committed to enhancing 
# 

utilization have a responsibility to train decision makers and information 
users in evaluation processes and the uses of information! By training 
stakeholders in evaluation methods and processes, the evaluator fs looking 
to both short-term and long-term utilization. Making decision makers more 
sophisticated about evaluation can contribute to greater use of evaluation 
datayand evaluation processes over time. 

s 
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An eighth premise is that there are a variety of ways in which 
evaluation processes and findings are used, a point noted earlier. 
Evaluations can directly influence major, specific decisions. Evaluations 
can be used to make minor adjustments in programs. Decision makers can, 
and do, use evaluations to reduce uncertainty, enlarge their options, 
increase control over program activities,' a?td increase their sophistication 
about program processes. Sometimes evaluations have more of a conceptual 
impact , i.e., they influence how stakeholders think about a program, rather 
than an instrumental impact , i.e., evaluation utilization manifested in 
concrete actions and explicit decisions. A broad view fof utilization 
reveals multiple layers of impact over varying amounts of time. All of 
these kinds of utilization are important apd legitimate form a 
utilization-focused evaluation perspective. This view of utilization aVso 
broadens tire notion-of evaluation impact to include use of the entire 
evaluation process as a stakeholder learning experience, not just use of 
the findings in thefinal report. The relative valuae of these different 
kinds of utilization can only be judged in the context of a specific 
evaluation. There is no universal hierarchy where some kinds of use are 
always more valuable, 

A ni^nth premise is that attention to Utilization involves financial 
> and staff time costs that are far from trivial. The benefits of these 
costs are manifested in greater utilization. These cost should be made 
explicit in evaluation proposals and budgets so that utilization efforts 
are not neglected for lack of resources. 

A tenth premise is that a variety of factors affect utilization. These 
factors include community variables, organizational characteristics, the 
nature of the evaluation, evaluator credibility, political considerations, 
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and resource constraints (^1 kin e_t aj_, 1979) In ^onducting a 
uti 1 ization-fo6used^v^Tuation, the evaluator attempts to be sensitive to 
and aware of how thes,e various factors affect the potential for 

! 

utilization. An analysis of the factors that may affect the usefulness of 

an evaluation should be undertaken jointly with statehol ders . early in the 

evaluation process. These factors, and their actual effects on 

utilization, are then monitored throughout the utilization-focused 

-\ 

eval uation process. 



TESTING THESE PREMISES IN PRACTICE 

The integrated nature of these premises with the underlying importance 

\ 

of the personal factor suggests to me the primary direction for/future 

> • 

research on utilization. That direction is to test these premises in an 

integrated way in real evaluations. By an integrated wa^ I mean that it is ^ 

important to look at utilization in teems of the related factors and 

prociesses that occur from beginning to end from a fairly holistic 

perspective. Studies that focus on one or two single, isolated factors are 

of little use fn extending our knowledge of real utilization processes. 

Laboratory experiments aimed at testing out what happens if people have 

more or 1 ess information , or some other single factor taken in isolation 

from. real world settings, *is also of little use. The premises stated above 

mean that utilization research must necessarily be holistic and must look 

at entire evaluation processes from beginning to end. There is no other 

way to test out these premises because they are not subjecc to simple 

t 

manipulation of operational variables. 

By way of illustrating how such tests can proceed, I would like to 
describe a recent equation project which employed a utilization-focused 
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project. That project is the Caribbean Agricultural Extension Project for 



evaluation perspective and permitted what has been in niy experience the 
most comprehensive test of utilization-focused evaluation in a real world 

which I act as Project Director. Thus, my account of the external 
evaluation of that project and its use is necessarily subjective. However, 
even as this paper is being written the chair of this external evaluation^ 
isfal so independently reviewing the utilization process. 

A CASE EXAMPLE IT UTILIZATION » 

« 

The Caribbean Agricultural Extension Project is a U.S.AID-funded 
project aimed at improving national agricultural extension services in 
eight Caribbean countries. With staff from the University of Minnesota and 
the University of the West Indies, the project has involved organizational 

development work with/key officials in the eight countries, providing 

. . </ 
in-service training for extension staff, and providing equipment including 

vehicles, office equipment, and agricultural equipment. 

The project was designed based on 1-1/2 years of needs assessment and 
planning. The assessment and p\anning included establishing an advisory % 
committee in each country as well as a regional advisory committee made up 
of representatives from all eight participating countries and other 
organizations involved in agricultural development *in the Caribbean. 

The contract for implementing the project with all key participants 
was signed in January, 1983. In April, 1983, a meeting of the Regional 
Advisory Committee was held with a team of external evaluato^s. The* 
external evaluators were chosen to represent the major constitu^p^es of 
the project, these being (1) U.S. AID, (2) the University of the West » k 

Indies, and (3) the Midwest Universities Consortium for InternatTO/ial 

i; 1 
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Activities (HUCIA) for which the University of Minnesota was the primary 
representative. Each of these three prime constituencies named one of the 
evaluators. The fourth evaluator wA*-^hosen for his stature in the field 
of evaluation, because of his comn/itment to user-oriented evaluations, and 
because^he was neutral from the point of view of the other three 
constituencies. He was made chair of the evaluation team so as to 
represent neutrality in collaboration with the three evaluators who had 
been named by specific constituencies as able to represent their points of 
view. 

Prior to designing the evaluation, the evaluators met with 
representatives each of these constituencies separately, including the 
funding sourcaJLS. AID.. - At the April meeting of the Regional advisory 
group, the evaluators focused three days of discussion on the criteria 
which could be used to determine if the project had been successful. These 
criteria constituted a set of questions and primary outcomes, but were not 
quantitative indicators. Based on those discussions the evaluators 
reviewed design possibilities with the fifty participants in that regional 
advisory meeting. The details of the design were then worked out with 
specific representatives of the project staff and U.S. AID. 

The evaluation design included several different foci. The project 
staff organized all of their required reporting around the evaluation 
design. The work plan /of projectf staff was also developed based on the 
Evaluation elem£q£s and staff meetings routinely reviewed the elements of 
the evaluation as a way of directing implementation and focusing on those 
outcomes which were primary from the point of V/ieWsOf the project and the 
evaluation. Members of the evaluation team were sent monthly and 
quarterly reports based on the elements of the evaluation. For example, 
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the first element in the evaluation design was that in each country s 






national agricultural extension planning committee be operating and 






involved in providing direction to the extension service in its country. 






All staff meetings began by reviewing the progress of national planning 






committees and all monthly and quarterly reports include information on the 






activities and progress of national committees. In addition, the minutes 






of the national planning committees were provided to the evaluators. In 






the actual data collection phase the evaluators conducted interviews to 






gather firsthand information about the operations and activities of the trie 






national planning committees. The point here is that program 






implementation and evaluation were synchronized from the beginning of each. 






More importantly, the evaluation process had a major impact on improving 






program implementation from the very beginning by focusing program 






implementation. The evaluation provided a framework for program planning 






and .reporting that provided focus to staff activities. This focus became 






more important as the project moved forward and staff encountered many 






opportunities to be diverted from those primary foci. However, having 






organized the project work plan, staff meetings, and reporting around the 






key evaluation elements, the evaluation contributed substantially to 






keeping staff efforts from being ^i^erted into other areas or activities 






which would have taken away from the primary purpose of the project. This 






is an example of utilization of evaluation processes for program 






improvement. 






Data collection and reporting were carefully timed to provide critical 






^ information for refunding decisions. Working backwards from the project 






completion date, a time schedule for data collection and reporting was 






developed which would make sure that the information was available when the 
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decision about refunding and future project activities was to be made. 
This was a major break with U.S. AID tradition. Indeed, the evaluators and 
project staff had some difficulty helping U.S. AID personnel understand why 
the evaluation was taking place so early in the life of the project, early 
from their perspective. Traditionally, U.S. AID evaluations occur after a 

4 1 

project is completed to provide a mandatory report on project impacts. 
That means that the traditional U.S. AID evaluation is presented six months 
to a year after a project is terminated or a new funding decision has been 
made. There is i possibility of the evaluation playing a role in that 
decision making. It was unprecedented for U.S. AID to get an evaluation 
report, at least one that was more than cursory, at the tiftie of a funding 
decision. In this case operational project funding would end in September, 
1985. Given the lengthy funding process of U.S. AID a decision for 
additional funding and activities would have to be made by December, 1984, 
to do the paperwork to keep the project alive. Thus, a meeting of the 
Regional Advisory Group was scheduled for November > 1984, to focus on the 

4 

evaluation findings. This meant that the report would have to be ready by 
that time so the data collection would have to take place in the Summer of 
1984, only 1-1/2 years into project implementation and only a year after 
the initial design, fully a year ahead of the operational project 
completion date. Clearly, such an evaluation could not be definitive about 
project impacts since data collection would take place well before project 
completion, but a definitive data collection effort would not be available 
at the time the decision was to be taken. Data collection did occur in 
June of 1984 and the evaluation report was ready for the Regional Advisory 
meeting in November, 1984. Prior to that critical November Regional 
Advisory Committee Meeting, the evaluators met separately with project 
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staff to provide informal feedback about evaluation findings and with 
U.S. AID to provide informal feedback and discussion of potential future 
funding. In both cases those informal meetings were critical. 

The first informal meeting occurred immediately after data collection 
in June, 1984. The .evaluators, who had been gathering data in different 
locations, met together to review their findings and divide the writing 
tasks. Following that session together, the evaluators met with the 
project director to review major findings. 'Those findings included a 
confirmation of the overall successes of the project, the high degree of 
support fjor project activities among the participating countries, and 
identification of areas of weakness. The areas of weaknejs^incl uded 
insights which had escaped the attention of project staff. The staff 
immediately began to correct those weaknesses, two of which required 
assistance from outside and one of which brought a new focus to 
implementation activities. A month later one representative of the 
evaluation team met with the project staff in their full staff meeting and 
reviewed the evaluation findings. It was at that staff meeting that 
activities were reoriented to direct attention to identified weaknesses. 

Following the staff .meeting, the evaluator who had been selected by 
U.S. AID met with U.S. AID officials to informally report initial findings. 
At that meeting the question of future funding arose. The director of the 
funding agency, U.S. AID, had been present in the initial meeting with the 
evaluators where important questions were identified. He now put those 
questions to the evaluator again with special reference to future 
activities. The evalua.tor was ab-le to directly address this ^question with 
high credibility and with concrete data. It has subsequently been reported 
to me independently by several U.S. AID staff that this informal feedback 
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was critical because the director of the funding agency was not predisposed 

to continue funding for the project. The evaluation report made it clear 

that the project was effective, was having an impact, but that further 

funding and activities would be necessary and justifiable to 

institutionalize short term successes and guarantee long term success and 

long term ef recti veness. With the November, 1984, Regional Advisory 

meeting already scheduled, and with- the informal evaluation results having 

been reported to the funding agency, I wrote to U.S.AID asking for them to 

take a position on their openess to future funding, I asked for a^response 

prior to the November, 19^4, meeting of the Regional Advisory group^ince 

the delegates to the Regional Advisory Group would need to know AID's 

position as a cofitext for their discussions of the evaluation. Prior to 

> 

that meeting U * S. AID indicated that, they had reviewed the evaluation and 
were inclined to continue funding activities. They therefore invited 
project staff and the Regional Ad vi sory group to submit a continuing 



Regional Advisory meeting. At that meeting the evaluators reviewed overall 
findings and different ways in which the report could be used for local 
purposes as well as regional purposes. Delegates to the meeting reviewed 
the executive summary and commented on its accuracy* They then adopted a 
resolution accepting the evaluation report as generally accurate, fair, 
thorough, and balanced. They suggested that project staff use the 
evaluation findings as a basis for future activities and a new propbsal to 
U.S.AID. They discussed major new directions suggested by the evaluation 
findings. They brought to bare on those discussions other information and 
their own experiences, and subsequently adopted resolutions identifying the 



proposal . 



The published evaluation report was completed 
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major components that should be included in continuing activities. 

In the interim between the data collection, informal feedback and the 
formal November review of the evaluation, project staff had made major 
progress in overcoming the weaknesses identified in the preliminary 
feedback. In addition, through the grapevine, the fact that the evaluation 
report would show substantial progress and major successes was communicated 
throughout the region. Project staff and U.S. AID had the opportunity to 
comment on draft copies of the report before it was published to guarantee 
accuracy and ,so as to know details of what the report would say prior to 
its publication. 

Thus, the evaluation had a major impact on project implementation. It 
had a major corrective effect in reorienting the project a year and a half 
into implementation so as to correct weaknesses that had emerged during 
that time and to more directly focus on some areas that were being 
neglected. Finally, the evaluation had*# major impact on the decision to 
continue project funding. 

The evaluation direct costs were approximately $100,000 out of a total 
project budget of $5.4 million dollars. This is under two percent of the 
project budget. 

While the details of this evaluation are skimpy to preserve space, all 
of the premises of utilization-focused evaluation were followed in this 
evaluation and the result was a high level of use. Of course, this design 
does not permit one to make causative statements about the relationship 
between what was done and what subsequently occurred. However, there is no 
question among the nine project staff members nor the U.S:AID officials 
that both the evaluation processes and outcomes made important differences. 
Likewise, the resolution adopted by the Regional Advisory group made it 
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clear that they had learned from the evaluation both about the project and 
about how evaluations ought to be conducted. ^ 
Issues Needing Clarification 

This discussion leads me to five issues that continue to create 
confusion in the field. At the recent ERS-ENet meetings in San Francisco I 
found evidence that these issues are alive and well. In some cases it 
seems to me we can put them to^rest. In- other cases new work is needed. 
These issues are: ^ 

JL 

1. What is the relationship between quantity and quality of 
interaction between evaluators and decision makers? 

2. How does heavy involvement of stakeholders ir an evaluation 
affect methodological quality? 

3. What, if any, is the hierarchy of desired impacts from an 
evaluation i.e., is immediate action a "greater impact" than 
long term effects on program thinking and conceptualization? > 

4. Is there a hierary in terms of the parts of an evaluation 
that ought to be used, i.e., is use of findings more important 
than use of the process, and are both more important than use 
of the evaluator? 

5. Is predetermined use for predetermined users more important 
'than unintended use in unintended ways? 

Permit me- to elaborate on the background which gives rise to these 
questions. The importance o.f the personal factor and its manifestation in 
"stakeholder-based evaluations" has led to confusion about the nature of 
evaluator involvement with key decision makers and information users. This 
is sometimes called "the stakeholder assumption." 

The "stakeholder assumption" is the idea that key people who have a 
stake in an evaluation should^ be actively and meaningfully involved in 
shaping that evaluation so as to focus the evaluation on meaningful and 
appropriate issues, thereby increasing the likelihood of utilization. A 
consultative evaluation approach is based on the stakeholder assumption. 
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In recent years, as evaluators have become increasingly concerned about 
utilization, the stakeholder assumption has received widespread attention^ 
Doubts have been raised about the validity of the assumption. Nick Smith, 

for example, president of the Evaluation Network during 1980, wrote in his 

— 

column in the Evaluation Network Newsletter that the assumption was being 
accepted^wtthout sufficient empirical evidence to support the supposed 
relationship between stakeholder involvement and utilization of findings. 



Although this (the stakeholder assumption) appears to be a 
widely held belief, no-pne has bothered to test it empirically. 
From a recent 16-state 'study of local district school accredi- 
tation evaluations, I have found that data from school board 
members and administrators wi th* f i rst-hand experience in such 
evaluations do not agree with this assumption. These individuals 
do not want to be personally more involved in such studies, nor 
do they believe that their involvement will make the evaluation 
results more useful to them. In fact, for these school board v 
members and administrators, tfte correlation between their 
judgments of a past evaluation's utility to them was only 0.3, 
while there was a 0.7 correlation between their judgments Qf the 
evaluation's quality and its utility. Hardly strong support for 
the considerable effort now being expended at the local, state, 
and federal level to increase the involvement of various groups in 
evaluation (Smith, 1980: 39). 

Smith's doubts about the validity of the stakeholder assumption 
provide an opportunity to clairfy my own interpretations about what the 
stakeholder assumption means in practice. His critique includes some 
common misconceptions about the collaborative approach to 
utilization-focused evaluation which give rise to the issues outlined 
above. 

First, there is the question of the nature of the relationship between 
stakeholder involvement and utilization. Smith states the relationship as 
a "necessary" condition. In the sentence preceding the excerpt quoted 
above, he said he was addressing "the currently popular assumption that 
increased involvement of clients and decision makers in evaluation 




activities will necessarily result in increased utilization of evaluation 
findings" (Smith, 1980: 39, emphash added). From my point of view, the 

stakeholder assumption is somewhat overstated by Smith. I have never 

i 

suggested^ or heard others suggest, that increased stakeholder involvement 
in an evaluation will necessarily, result in increased utilization. Nothing 
one can do, as near as I can tell, will guarantee utilization. 

Second, there is a hint of a tradeoff in Smith's skepticism implying 
that one must choose between stakeholder involvement and high quality data. 
Many evaluators assume that methodological rigor will inevitably be 
sacrificed if nonscientists collaborate in making methods decisions. This 
need not be the case. The ideal expressed in the new standards includes 
both utility and accuracy. Other research confirms Smith's findings that 
decision makers are concerned about "quality" of data, but quality includes 
both "truth tests" (accuracy) and "utility tests," the latter being a 
concern for relevance and applicability (Weiss and Bucuvalas, 1980). 

Third, Smith's point is directed entirely at the quantity of 
stakeholder involvement in an evaluation. The variable "level of 
involvement" is somewhat ambiguous, but the implication is that it refers 
to amount of involvement in terms of time. In Contrast, ,the emphasis in 
utilization-focused evaluation is on careful selection of the people with 
whom one works and the qua! ity of the evaluation group^ process. 
quant i ty of interaction time is often inversely related to quality . 

Fourth, while I 'expect Smith is cdrr^tf in saying that there is a 

dearth of empirical evidence that "increased involvement" (if he means 

greater quantity) leads to greater utilization, there is substantial 

evidence that hi gh-qual ity .invpl vement of the right people contributes 

tup 

substantially to utilization. Jhe massive diffusion of innovation 
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literature (e.g., Havelock, 1971, 1973; Rogers and Shoemaker, 1971) is 
replete with relevant empirical evidence. The formal* organizations, 

participatory management, and smalt group literatures in psychology and \ 

\ 

.sociology provide substantial data relevant to this point (e.g., Hage and \_ ^ 
Aiken, 1970; Bennis, 1966; Azumi and Hage, 1972; Bertnis et al., 1976; 
Argyris, 1972, 1974, 1976). These literatures document with empirical 
evidence the proposition that people are more likel^xto accept and use 
information, and make changes based on information, when they are 
personally involved in and have a personal stake in the decision making 
processes aimed at bringing about change. Most directly, there is a 
growing evaluation and pol icy'analysis literature— )in empirical 
1 iterature—that supports the proposition that utilization of evaluation is 
enhanced by high-quality stakeholder involvement in and commitment to the 
evaluation process (e.g., Fairweather et al., 1974; Weiss, 1977; Patton, 

\ 

1978; Alkin et al., 1979; Braskamp and Brown, 1980; Stevens and Tornatsky, 
1980; Lynn, 1980; Dickey, 1981). ' 

Fifth, evaluators should not expect much initial enthusiasm among* 
stakeholders far the idea of participating actively in a research process. 
Past experiences are not likely to have been very positive. Most 
stakeholders are quite happy to leave evaluation to evakiators. They're 
also quite happy to ignore the resultant evaluation findings. Like a child 
who wants to avoid bad-tasting medicine (medicine, by definition, being bad 
tasting), stakeholders would typically prefer to avoid being subjected to 
distasteful doses *Of evaluation (evaluation, by definition, being 



distasteful), even if they believe it's cjaod for them. The evaluation 
practitioner like the medical practitioner, must often cajole and 



otherwise persuade stakeholders to do what ought to be done. ^Getting 
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cooperation and participation- has to be workeji at. Initial resistance is 

no reason to fall back on traditional patterns v x)f operating alone, at least 

\ 

not if the evaluator is really committed to utilization. In my experience, 

if stakeholders won't get involved at the beginning^of an evaluation, they 

pfobably won't pay it much heed at the end. 

If utilization is. viewed as use of the evaluation process and not just 

as final findings, then stakeholders must be involved in the entire process 

for the process to have the most impact. Much of the impact of evaluations 
* 

on stakeholders comes through personal engagement in the difficult 
processes of goal s ;clari fication , issues identification, operational i zi ng 
outcomes, matching research design ^o program design, jdetermining sampling 
strategics, Organizing data collection, interpreting results, and drawing 
conclusions. These processes take stakeholders through a gradual awakening 
to program complexities and realities, an awakening that contains 
understandings and insights that \|rNl find their way into program 
developments over time, only some of which wilJ be manifested in concrete 
decisions. Utilization begins as soon as stakeholders become actively 
involved in evaluation because that involvement, properly facilitated, 
forces them to think about program priorities -and realities. The 
stakeholder assumption, thW, includes the expectation that stakeholders 
need to expend time and effort to figure out what is worth doing in an v 
evaluation; they need help in focusing on worthwhile questions^and they 
need to experience the full evaluation process if that process, which is 
really a learning process," is to realize its potential, i^ulti -1 ayered* 
effpets. S 

To summarize, there are several key issues that emerge from 
discussions of the stakeholder assumption and research on the personal 
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factor: 

1. What is the relationship between quantity and quality 
of interaction between evaluators and decision makers? 

2. How does heavy involvement of stakeholders in an 
evaluation affect methodological quality? 

3. What, if any, is the hierarchy of desired impacts 
from an evaluation i.e., is immediate action a 
"greater impact 1 ' than long term effects on program 
thinking and conceptualization? 

4. Is there a hierarchy in terms of the parts /f an 
evaluation that ought to be used, i.e., is 'use of 
findings more important than use of the process, 

and are both more important than use of the evaluator? 

* 5. Is predetermined use for predeterny-ned users more 
important than unintended use in unintended ways? 

These issues emerge from the implicit hierarchies and values that seem 
to me to be present in the current research on evaluation. These^are not 
simply research^issues. Rather, they have to do with how the research on 
utilization ought to be conceptualized for future work. 

Before closing (this paper, I would like to raise one additional issue 
for discussion. That issue is the misutil ization of evaluations. 



MISUTILIZATION 

If it is not already clear to the reader, let me make it absolutely 
clear: I believe that we already know enough about how to increase the 
utilization of evaluations that the immediate task is acting on what we 
know and evaluating those actions rather than doing further isolated 
research on utilization. In so acting, and in order to be accountable, 
evaluators ought to document their experiences in using what^we know. In 
presenting my recent experiences with the Caribbean Agricultural Extension 
Project as a test of utilization-focused evaluation premises, I have 
indicated in very brief fashion how such documentation might take place. 
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For my part, however, I don't look for any major new breakthroughs in 
research on the utilization of evaluation. I'm satisfied that if We 
actually begin tp use what we already know, we could make a significant 
difference in evaluation practice. 

I have become, however, increasingly concerned about problems of 
misuti 1 ization . The most common criticism I hear of utilization-focused 
evaluation is that it co-opts evaluators. This cooptation reduces 
evaluation credibility, neutrality, and signi f icance* 

In our concern with and focus on ways of increasing the utilization of 
evaluation, I agree with those who worry that we have neglected 
misuti 1 ization . As I do workshops and travel around the country talking 
with people about evaluation, I hear increasingly about cases of abuse and 
misuse. As I've thought about this, I'd like to share some preliminary 

observations by way of generating additional discussion on this important 

> 

issue. 

1. Mi suti 1 ization is not at the opposite end of a continuum from 
utilization. There are really two dimensions here. One dimension is a 
continuum from nonutil ization to utilization, A second continuum is 
non-misuti 1 ization to misutil ization. Studying misutilization is quite 
different from studying utilization* 

2. Having conceptualized two separate dimensions, it is possible to 
explore the relationship between them. Therefore, permit me the following 
proposi tion: as utilization increases, misutilization will also Increase . 
It seems to me that when people ignore evaluations, they ignore their 



potential uses as well as abuses. As we successfully focus greater 
attention on evaluation data, and as we increase actual use we can also 
expect there to be a corresponding increase in abuse, often within the same 
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evaluation experience. 

3 #- Misuse can be either intentional or unintentipnal . Unintentional 
misutilization cfcn be corrected through the processes aimed at increasing 
appropriate and proper utilization. Intentional misutilization is an 
entirely different matter to which, it seems to me, we have paid very 

■s 

little attention except to say it shouldn't happen. In. terms of incidence 

, ) 
and prevalence, I have no clear notion of whether unintentional or 

intentional misuse is more common. > 

4. A comprehensive approach to the study of misutilization might well 

be guided by the same six honest serving men who framed my initial 

discussion of this paper of utilization. 

1 . What is misused? / 

2. Who misuses evaluation? 

3. When is evaluation misused? 

4. How is it misused? 

5. Where is misused? 

6. Why W it misused? 

What better way to approach dishonesty and misuse than by mobilizing 
these six honest serving men in the service of appropriate evaluation 
uti 1 i zation? / 

/ y 

\ 
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